Skip to content

feat: add PaperReader v0.2 plugin — 超文本递归阅读器 #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Andyduck-ops wants to merge 6 commits into
base: main
Choose a base branch
from
Open

feat: add PaperReader v0.2 plugin — 超文本递归阅读器 #194

Andyduck-ops wants to merge 6 commits into from

Conversation

@Andyduck-ops
Copy link

@Andyduck-ops Andyduck-ops commented Feb 7, 2026

PaperReader v0.2 — 超文本递归阅读插件

功能概述

将超长 PDF/文档转为可递归阅读的分块工件，适用于学术论文、技术报告、书籍章节、法律文书等各类长文档。

架构

  • L0 解析层：MinerU 云端 API 高保真解析（公式/表格/图片/多栏/扫描OCR），不可用时自动降级到 pdf-parse
  • L1 切分层：章节感知切分 + Meta-Header 注入 + 10-20% overlap + tiktoken 精确计数
  • L2 递归逻辑层：Skeleton 骨架提取 / Rolling Context 深度阅读 / 合并综合
  • L3 存储层：Obsidian 友好的 Markdown 目录结构

命令

命令 功能
IngestPDF PDF → Markdown → 章节感知 chunks
ReadSkeleton 从目录/摘要/关键章节生成 Global Map
ReadDeep 带 Rolling Context 的递归摘要 → Round-1 笔记
Query 检索式问答（关键词匹配 + 章节权重）

文件清单

Plugin/PaperReader/
├── PaperReader.js          # 主入口（stdin JSON → 路由 → stdout JSON）
├── plugin-manifest.json    # VCP 插件清单
├── config.env.example      # 配置示例
├── README.md               # 使用说明
└── lib/
    ├── mineru-client.js     # MinerU 云端 API 适配器
    ├── pdf-parse-fallback.js # pdf-parse 降级封装
    ├── ingest.js            # 统一解析入口（优先 MinerU，自动降级）
    ├── chunker.js           # 章节感知切分器
    ├── llm.js               # LLM 调用封装
    ├── skeleton.js          # 骨架提取（Global Map 生成）
    ├── deep-reader.js       # Rolling Context 深度阅读
    └── query.js             # 检索式问答

依赖

  • adm-zip（新增，用于解压 MinerU 返回的 zip）
  • axios、pdf-parse、@dqbd/tiktoken、dotenv（已有）

配置

复制 config.env.example 为 config.env，填入 MINERU_API_TOKEN（可选，不填自动降级）和 PaperReaderModel。

@Andyduck-ops
feat: add PaperReader v0.2 plugin - universal document recursive reader
531f7e3 
New plugin: Plugin/PaperReader
- MinerU cloud API for high-fidelity PDF parsing (formulas/tables/images/OCR)
- Auto-degradation to pdf-parse when MinerU unavailable
- Section-aware chunking with Meta-Header injection
- Rolling Context deep reading with auto-compression
- Keyword-based query with section weighting
- Supports academic papers, technical reports, books, legal documents, etc.
@Andyduck-ops
fix: correct MinerU API model_version and upload headers per official…
9fdeda2 
… docs

- model_version: use 'pipeline' (default) or 'vlm', not 'hybrid-auto-engine'
- Remove Content-Type header from PUT upload (per MinerU docs)
- Add MINERU_MODEL_VERSION config option
- Clarify config.env.example: Bearer Token, not Access Key/Secret Key
@Andyduck-ops
fix: MinerU OSS upload 403 + pdf-parse v2 API compatibility
499bd1a 
- mineru-client.js: replace axios.put with native https.request for OSS
  presigned URL upload (axios auto-adds headers that break signature)
- pdf-parse-fallback.js: rewrite for pdf-parse v2 API
  (PDFParse class + Uint8Array + getText/destroy)
@Andyduck-ops
fix: return content to model + 429 retry with exponential backoff
3614ecd 
- PaperReader.js: ReadSkeleton/ReadDeep now return actual text content
- llm.js: add 429 exponential backoff retry (5 attempts, 3s/6s/12s/24s)
- deep-reader.js: add 1.5s inter-chunk delay to prevent rate limiting
@Andyduck-ops
fix(PaperReader): fix LLM URL resolution, add concurrency & caching
dce4176 
- lib/llm.js: auto-resolve localhost URL to correct VCP port + path;
  add error classifier (rate_limit/auth/timeout/network); add traceTag logging
- lib/deep-reader.js: batch-internal concurrency via Promise.all;
  full cache (skip if Round_1_Summary.md exists);
  incremental cache (skip already-summarized chunks)
- PaperReader.js: IngestPDF cache (skip if manifest+meta exist);
  route logging; passthrough forceReparse/forceReread params
@Andyduck-ops
Copy link
Author

Andyduck-ops commented Feb 7, 2026

Hotfix: LLM URL 解析 + 真并发 + 三层缓存

本次 push 修复了以下问题：

🔧 Bug Fix: LLM 429 错误（根因）

  • lib/llm.js 的 API_URL=http://127.0.0.1:3000 指向 VCP 管理面板，不是 API 端点
  • 新增 resolveApiUrl() 自动修正为 http://127.0.0.1:{PORT}/v1/chat/completions
  • API Key 优先级链：PaperReaderApiKey > Key > API_Key

⚡ 性能: 真并发

  • batchSize=10 现在是真正的 10 路并发（Promise.all），不再是串行
  • batch 间按序合并 rolling context 保持连贯性

💾 缓存机制

  • IngestPDF 缓存：已解析的 PDF 不重复调用 MinerU
  • ReadDeep 全量缓存：已完成的深度阅读零 LLM 调用直接返回
  • ReadDeep 增量缓存：中断恢复时跳过已完成 chunk
  • 支持 forceReparse / forceReread 参数强制刷新

📊 可观测性

  • 请求入口 / 路由命中 / chunk 进度 / 错误分类日志
  • 错误分层：rate_limit / auth / timeout / upstream_5xx / network / unknown

✅ 测试通过

  • ReadDeep 并发 ✅ | 缓存命中 ✅ | 缺参校验 ✅ | URL 自动修正 ✅

@Andyduck-ops
feat(PaperReader): v0.4 — 统一自适应阅读引擎
f8cda11 
- 新增 Read 命令: Survey→Triage→DeepDive/Skim→Audit→Synthesize
- 新增 Triage 分诊模块 (lib/triage.js): 注意力分配 + 拓扑排序
- 新增 Skim 模式 (lib/skim-reader.js): 轻量扫读 + upgrade 检测
- 新增 Auditor 审核员 (lib/auditor.js): 去污染独立审核 + PatchContext
- 新增 ReadingState 持久化 (lib/reading-state.js)
- Bug修复: Promise.allSettled 容错, 增量缓存 readMode 校验
- Bug修复: expandToChunkPlan skip 节点统计, O(n) 环检测
- 并发调优: BatchSize 2→5, MaxConcurrentLLM 3→5, ChunkDelay 1500→800ms
- 更新 README, config.env.example, plugin-manifest.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Andyduck-ops